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PROBLEM 


Need  to  be  able  to  validate  medical  models 
using  DoD  Deployable  Medical  System 
(DEPMEDS)  Patient  Conditions  (PCs)  with  real 
world  data  documented  with  I  nternational 
Classification  of  Disease,  9th  Revision,  Clinical 
Modification  (ICD-9-CM)  codes 


23  J  une  2005 


Center  for  AMEDD  Strategic  Studies  for  73rd  MORSS 


2 


OBJ  ECTI VE 


•  Evaluate  quality  of  mapping  between  these 
two  medical  classification  systems 

-  Real  World/ Existing  Medical  Data:  International 
Classification  of  Disease,  9th  Revision,  Clinical 
Modification  (ICD-9-CM)  diagnosis  codes 

-  Medical  Modeling  and  Simulations:  DoD 
Deployable  Medical  System  (DEPMEDS)  patient 
condition  (PC)  codes 
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PATI  ENT  CONDITIONS  (PCs) 

•  Each  PC  represents  a  group  of  patients  with 
similar  medical  conditions,  therefore,  similar 
treatment  requirements 

•  Total  of  389  PCs  (codes  numbered  1  -  440) 

-  318  PCs  for  conventional  warfare  (codes  1-350) 

•  96  for  disease 

•  146  for  non- battle  injury  (NBI ) 

•  187  for  wounded  in  action  (Wl  A) 

-  75  PCs  for  nuclear,  biological,  and  chemical 
warfare  (codes  351-440) 
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PROBLEMS  WITH  PCs 


•  PCs 

-  Do  not  identify  specific  clinical  diagnoses  or  procedures 

•  Associated  Treatment  Brief 

-  General  in  nature 

-  Data  based  on  subject  matter  expert  opinion 

•  Associated  frequencies,  treatment  time,  hospital 
length  of  stay,  and  OR  time 

-  Based  on  subject  matter  expert  opinion 

-  Not  directly  derived  from  live  data,  such  as  SADR/SI  DR  with 
ICD-9-CM  diagnosis  and  procedure  codes 
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MODELS  AND  SI  MULATI ONS 


•  Models  and  simulations  are  increasingly  important  to 
DoD  medical  community 

-  Requirement  estimation  for  casualty  care 

-  Patient  care 

-  I  ntegration  of  medical  into  line  models  and  simulations 

•  Planning  factors  used  in  requirements  estimation 
models  and  simulations  based  on  SME  data 
associated  with  DEPMEDS  PCs 

-  Treatment  time  (below  level  III) 

-  Length  of  stay  (by  bed  type,  level  1 1 1  and  above) 

-  OR  time 

-  Probability  of  RTD,  Death,  Evacuation 
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PC  TO  I CD9  PROJ  ECT  DESCRI PTI  ON 


•  Goal:  Determine  if  mapping  between  coding 
systems  possible,  so  medical  model  results  can 
be  validated  with  real  world  data 

•  Data  from  Army  Graduate  Management  project 

-  Three  Certified  Coders  given  PC  Treatment  Briefs 

-  Each  coder  independently  assigned  all  applicable 
I CD-9-CM  diagnosis  codes  to  each  PC 

-  Each  coder  identified  "key"  diagnosis  codes 

-  I  nitial  set  -  40  PC  codes 

-  Final  set  -  389  PC  codes  ( including  initial  40) 

•  Evaluate  reliability/consistency  of  coding 
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RATER  RELIABI  LITY 


•  Interrater  reliability  measures  agreement,  or 
consistency,  in  judgments  by  two  or  more 
individuals  assessing  same  information 

•  I  ntrarater  reliability  measures  consistency  in 
judgment  by  an  individual  assessing  same 
information  multiple  times 
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RATER  RELIABI  LITY 


•  Methods  of  measuring  rater  reliability  when 
outcomes  are  nominal 

-  Percent  agreement:  Ratio  of  #  times  2  raters 
agree/ total  ratings  performed  (0-100%) 

-  Cohen's  kappa  statistic:  Chance-corrected 
proportion  of  agreement  (Cohen,  1960) 

•  +  kappa  4  Agreement  better  than  chance  (max=+1.00 

and  occurs  if  total  agreement) 

•  0.0  4  Agreement  at  chance  level 

•  -  kappa  4  Agreement  worst  than  chance  (min=-1.00) 
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KAPPA  STATI STI C 


#  exact  agreements  - I  freq  agreements  expected  by  chance 
#  possible  agreements  -  I  freq  agreements  expected  by  chance 


Example 


Coder  1 

864 

884 

894 

959 

Missing 

^chance 

864 

1 

¥ 

1 

0.25 

CM 

L. 

884 

r 

0 

Chance 

0.00 

a> 

~o 

894 

1 

¥ 

1 

Agreements  > 

0.25 

o 

O 

959 

1 

F 

1 

0.00 

Missing 

1 

1 

Possible 

0.25 

w 

1 

¥ 

1 

¥ 

1 

0 

1 

4 

<  Agreements 

0.75 

Exact  Agreements > 

2 

v  — 

2 

- 

0.75 

1.25 

0.3846 

l\ 

4 

- 

0.75 

3.25 
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CHALLENGES  I N  PERFORMI NG  ANALYSES 


•  Mapping  complexity 

-  Most  nominal  outcome  rating  schemes: 

•  Rater  chooses  single  response  per  observation 

•  10  or  fewer  response  categories  available 

-  PCs  mapped  to  standard  diagnoses: 

•  Rater  can  choose  several  responses  (diagnosis  codes) 
per  observation  (PC) 

•  More  than  10,000  response  categories  available 

-  Literature  searches  found  no  rater  agreement 
analyses  where  raters  could  choose  multiple 
responses  from  very  large  number  of  categories 
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CHALLENGES  I N  PERFORMI NG  ANALYSES 


•  Study  design 

-  Multiple  ICD-9-CM  diagnosis  codes  selected  for 
each  PC 

-  Coders  provided  minimal  instruction 

-  Data  not  checked  for  consistency  or  completeness 
at  time  of  collection 

•  kappa  calculation  by  SAS® 

-  SAS  computes  kappa  statistic  from  frequency  tables 

-  Tables  must  be  square  (both  raters  used  same 
categories) 

-  One  rater's  responses  form  rows,  another  rater's 
responses  form  columns 
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SOLUTI ONS 


•  Simplified  mapping  complexity 

-  Built  data  sets  with  consistent  record  layouts  and 
formats 

-  Converted  5-character  I CD-9-CM  diagnosis  codes 
to  3-character  codes  ( I CD9  codes) 

•  Performed  pairwise  analysis  of  3  coders 

-  Determined  correct  table  structure 

-  Created  square  tables  with  real  and  pseudo  data 

-  kappa  calculation  by  SAS 
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SOLUTI ONS 


•  Obtained  reliability  measures  from  3 
viewpoints 

1)  Mappings  of  individual  PC  codes  (amount  of 
agreement  in  assigning  I CD9  codes  to  a  PC  code) 

2)  Identical  mappings  of  individual  PC  codes 
(agreement  is  defined  as  assigning  same  set  of 
I CD9  codes  to  a  PC  code) 

3)  Mappings  without  regard  to  individual  PC  codes 
(what  proportion  of  time  did  coders  map  to  same 
ICD9  code  across  all  PC  codes) 
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EXAMPLE  OF  REAL  &  PSEUDO  DATA 


RECORDS  WITH  ASSIGNED  WEIGHTS 


PC 

CoderA 

CoderB 

CoderC 

wt 

Real 

0001 

X 

800 

800 

1 

0001 

X 

801 

801 

1 

0001 

X 

802 

X 

1 

0001 

803 

803 

803 

1 

0001 

X 

804 

804 

1 

0001 

850 

850 

X 

1 

Pseudo 

0001 

800 

800 

800 

IE-10 

0001 

801 

801 

801 

IE-10 

0001 

802 

802 

802 

IE- 10 

0001 

803 

803 

803 

IE- 10 

0001 

804 

804 

804 

IE-10 

0001 

850 

850 

850 

IE-10 

0001 

X 

X 

X 

IE- 10 
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EXAMPLE 


The  SAS  System 

- PC=0001 - 

The  FREQ  Procedure 
Table  of  CoderA  by  CoderB 

CoderA  CoderB 


Frequency 

X 

800 

801 

802 

803 

804 

850 

Total 

X 

IE- 10 

1 

1 

1 

0 

1 

0 

4 

800 

0 

IE- 10 

0 

0 

0 

0 

0 

IE- 10 

801 

0 

0 

IE- 10 

0 

0 

0 

0 

IE- 10 

802 

0 

0 

0 

IE- 10 

0 

0 

0 

IE- 10 

803 

0 

0 

0 

0 

1 

0 

0 

1 

804 

0 

0 

0 

0 

0 

IE- 10 

0 

IE- 10 

850 

0 

0 

0 

0 

0 

0 

1 

1 

Total 

IE- 10 

1 

1 

1 

1 

1 

1 

6 
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EXAMPLE 


Statistics  for  Table  of  CoderA  by  CoderB 
Test  for  Symmetry 


Statistic  (S) 

4.0000 

DF 

21 

Pr  >  S 

1.0000 

Kappa  Statistics 

Statistic 

Value 

ASE  95%  Confidence  Limits 

Simple  Kappa 

0.2941 

Sample  Size 

0.1558  -0.0113  0.5995 

=  6.0000000007 

Percent  agreement  =  #agreements/#ICD9  codes  compared  *  100% 

=  (2/6)  *  100%  =  33.3% 
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RESULTS 


•  Summary  results  are  presented  for  each 
viewpoint 

-  By  coder  pair  (AB,  AC,  BC) 

-  Overall  average 

•  Outcome  measures 

-  Mean  number  of  agreements 

-  Mean  number  of  comparisons  (e.g.,  average 
number  of  I CD9  codes  each  coder  assigned) 

-  Mean  percent  agreement  (mean  of  given  variable; 
not  based  on  ratio  of  mean  number  of  agreements 
to  mean  number  of  comparisons) 
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RESULTS 


-  Kappa  statistic 

•  Mean  (mean  of  given  variable;  not  based  on  chance 
corrected  ratio  of  mean  #  agreements  to  mean  # 
comparisons) 

•  SD  (standard  deviation  on  mean  kappa) 

•  Mode  (most  frequently  occurring  kappa  value) 

•  Q1  (1st  quartile  or  25th  percentile — value  at  or  below 
which  lie  lowest  25%  of  given  set  of  kappas) 

•  Median  (2nd  quartile  or  50th  percentile) 

•  Q3  (3rd  quartile  or  75th  percentile) 

•  95%  confidence  intervals  on  mean  kappa 
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RESULTS 


-  Kappa  statistic 

•  Mean  (mean  of  given  variable;  not  based  on  chance 
corrected  ratio  of  mean  #  agreements  to  mean  # 
comparisons) 

•  SD  (standard  deviation  on  mean  kappa) 

•  Mode  (most  frequently  occurring  kappa  value) 

•  Q1  (1st  quartile  or  25th  percentile — value  at  or  below 
which  lie  lowest  25%  of  given  set  of  kappas) 

•  Median  (2nd  quartile  or  50th  percentile) 

•  Q3  (3rd  quartile  or  75th  percentile) 

•  95%  confidence  intervals  on  mean  kappa 
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I NTERRATER  AGREEMENT  OF  3  CODERS 
I N  MAPPI NG  389  PC  CODES:  Viewpoint  I 
Mappings  of  I  ndividual  PC  Codes 


Coders 

Mean  # 
Agreements 

Mean  # 

ICD9 

Codes 

Compared 

Mean  % 
Agreement 

Kappa  Statistic 

Mean 

SD 

Mode 

Q1 

Median 

Q3 

AB 

1.6 

3.2 

68.7 

0.63 

0.40 

1.00 

0.33 

0.69 

1.00 

AC 

1.5 

3.0 

67.5 

0.60 

0.43 

1.00 

0.31 

0.69 

1.00 

BC 

1.4 

2.9 

65.6 

0.59 

0.42 

1.00 

0.26 

0.57 

1.00 

Overall 

1.5 

3.0 

67.3 

0.61 

0.42 

1.00 

0.31 

0.60 

1.00 
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I NTERRATER  AGREEMENT  OF  3  CODERS 
MAPPI NG  389  PC  CODES:  VI EWPOI  NT  1 1 
Identical  Mappings  of  Individual  PC  Codes 


Coders 

#  of  Perfect 
Agreements 

%  Agreement 

Kappa  Statistic 

Kappa 

95%  Confidence 

1  nterval 

AB 

188.0 

48.5 

0.48 

0.43 

0.53 

AC 

180.0 

46.6 

0.47 

0.42 

0.52 

BC 

177.0 

45.9 

0.46 

0.41 

0.51 

Mean 

181.7 

47.0 

0.47 
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I NTERRATER  AGREEMENT  OF  3  CODERS 
MAPPI NG  389  PC  CODES:  VI EWPOI  NT  1 1 1 
Mappings  Without  Regard  to  Individual  PCs 


Coders 

#  of 

Agreements 

#  of  1  CD9s 
Compared 

%  Agreement 

Kappa  Statistic 

Kappa 

95%  Confidence 

1  nterval 

AB 

602.0 

1224.0 

49.2 

0.45 

0.42 

0.49 

AC 

550.0 

1125.0 

48.8 

0.46 

0.42 

0.49 

BC 

502.0 

1104.0 

45.5 

0.41 

0.37 

0.45 

Mean 

551.3 

1151.0 

47.8 

0.44 
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RESULTS  FOR  5  PC  CODES  ACCOUINTTI  NG  FOR 
25%  OF  SI  MULATED  Wl  A  CASUALTI ES 


PC  Code 

Wl  A 

Mean  % 
Agreement* 

Kappa  Statistic* 

% 

Mean 

Min 

Max 

131  Wound  Lower  Leg  Open  Lacerated 

Penetrating  Perforating  With  Fracture  And 
Nerve  And/Or  Vascular  1  njury  Limb 
Salvageable 

7.29 

70.0 

0.64 

0.52 

0.69 

124  Wound  Thigh  Open  Lacerated  Penetrating 
Perforating  With  Fracture  And  Nerve 
And/Or  Vascular  1  njury  Limb 

5.30 

100.0 

1.00 

1.00 

1.00 

186  Multiple  Non- perforating  Fragment 

Wounds  Of  Skin  And  Soft  Tissue 

4.00 

55.6 

0.50 

0.25 

1.00 

048  Wound  Upper  Arm  Open  With  Fractures 
And  Nerve  1  njury  No  Vascular  1  njury  Arm 

3.98 

77.8 

0.71 

0.57 

1.00 

129  Wound  Lower  Leg  Open  Lacerated 

Penetrating  Perforating  Without  Fractures 
Not  Requiring  Major  Debridement 

3.84 

100.0 

1.00 

1.00 

1.00 

*  Viewpoint  I 

Center  for  AMEDD  Strategic  Studies  for  73rd  MORSS 
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Dl  SCUSSI  ON 


•  Only  moderate  reliability  outcomes  obtained 

-  Partly  due  to  minimal  instructions  to  coders 

-  Primarily  due  to  complexity  of  problem 

•  Not  1-to-l  correspondence  between  two  coding  systems 

•  389  possible  PC  codes  versus  about  10,000  ICD-9-CM 
diagnosis  codes  (or  1,000  3-character  ICD9  codes) 

•  Mixed  reliability  for  top  PC  codes  in  existing 
models 


23  J  une  2005 


Center  for  AMEDD  Strategic  Studies  for  73rd  MORSS 


25 


RECOMMENDATI ONS 


•  Redo  mapping  effort 

-  Provide  better  data  collection  directions 

-  Convert  from  diagnosis  codes  to  PC  codes 

-  Use  Delphi  method  to  improve  consistency 

•  Look  at  a  DRG-type  mapping  process 

-  Decision  tree  to  determine  PC 

•  Replace  PCs  in  models  with  another  system 
that  can  be  validated  and  periodically 
modified  by  real  theater  data 
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Questions? 
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